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ABSTRACT 


\j 

ISODATA,  a  novel  method  of  data  analysis  and  pattern  classification, 
is  described  in  verbal  and  pictorial  terms,  In  terms  of  a  two-dimensional 
example,  and  by  giving  the  mathematical  calculations  that  the  method  uses. 
The  technique  clusters  many- variable  data  around  pointB  in  the  data's 
original  hlgh-dlmenslonal  space  and  by  doing  so  provides  a  useful  descrip¬ 
tion  of  the  data.  A  brief  summary  of  results  from  analyzing  alphanumeric, 
gausslan,  sociological  and  meteorological  data  Is  given. 

In  the  appendix,  generalizations  of  the  existing  technique  to  cluster¬ 
ing  around  lines  and  planes  are  discussed  and  a  tentative  algorithm  for 
clustering  around  lines  Is  given.: 
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ISODATA,  A  NOVEL  METHOD 
OF  DATA  ANALYSIS  AND  PATTERN 
CLASSIFICATION 

by 

Geoffrey  H,  Bell  gnd  D&vid  J,  Hsill 
Stanford  Research  Institute 


I  INTRODUCTION 

In  this  paper  we  discuss  a  technique  for  dealing  with  problems  in 
whicli  the  data  are  inherently  described  in  many  dimensions,  where  each 
dimension  corresponds  to  a  variable  of  the  problem.  Such  problems  are 
very  commonplace,  and  many  which  are  customarily  described  in  terms  of 
a  small  (3  to  10)  number  ul  dimensions  are  in  fact  of  a  much  larger  di¬ 
mensionality,  but  have  been  simplified  in  order  to  allow  manipulation 
(and  description)  of  the  data.  Such  collapsing  of  the  problem  Is  often 
useful  and  suitable,  indicating  the  preponderant  importance  of  certain 
of  the  parameters.  However,  there  remains  a  class  of  problems  for  which 
such  collapsing  destroys  significant  Interrelations  between  the  parameters 
that  would  give  the  data  meaning. 

II  PATTERN  RECOGNITION  PREPROCESSING  AND  CIASSIFICATION 

The  area  of  research  labelled  "pattern  recognition"  consists  pri¬ 
marily  of  efforts  to  develop  techniques  capable  of  dealing  with  problems 
of  Inherently  high  dimension. 

Many  aspects  of  the  pattern  recognition  problem  are,  In  fact,  data 
analysis  called  by  a  different  name.  Realizing  this,  we  feel  free  to 
discuss  ISODATA  In  the  context  of  pattern  recognition  (which  Is  our  back¬ 
ground)  ,  although  others  might  prefer  "automatic  data  analysis"  as  the 
label  for  our  work. 

One  statistician,  John  W.  Tukey,  has  stated  that,  in  hla  mind,  data 
analysis  includes,  among  other  things:  "procedures  for  analyzing  data, 
techniques  for  interpreting  the  results  of  such  procedures,  ways  of  plan¬ 
ning  the  gathering  of  data  to  make  Its  analysis  easier,  more  precise  and 
more  accurate,  and  all  the  machinery  and  results  of  (mathematical) 

* 

Examples  of  such  problems  include  automatic  speech  recognition,  medical 
diagnosis,  alpha-numeric  character  recognition,  sociological  question¬ 
naire  analysis,  and  weather  prediction. 
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statistics  which  apply  to  analyzing  data 


„1 


Dr.  Tukey  goes  on  to  say  that  he  considers  data  analysis  as  a 
science  in  the  sense  that  it  has: 


a) 


intellectual  content 


b)  organization  into  an  understandable  form 

c)  reliance  upon  the  test  of  experience  as  the  ultimate 

standard  of  validity.  ~~ 

As  a  science  he  feels  that, 

a)  "Data  analysis  must  seek  for  scope  and  usefulness  rather 
than  security. 

b)  Data  analysis  must  be  willing  to  err  moderately  often  in 

order  that  inadequate  evidence  shall  more  often  suggest 
the  right  answer.  “  ™~ 

c)  Data  analysis  must  use  mathematical  argument  and  mathe¬ 
matical  results  as  bases  for  judgment  rather  than  as 
bases  for  proof  of  validity." 

We  feel  that  our  work  on  ISODATA  fits  this  description  of  data  anal¬ 
ysis.  Since  we  have  also  been  able  to  describe  ISODATA  in  the  terminology 
of  pattern  recognition,  we  feel  justified  in  relating  data  analyai#  to 
pattern  recognition.  Our  work  on  ISODATA- POINTS  has  concentrated  pri¬ 
marily  on  developing  the  algorithm  and  demonstrating  experimentally  that 
it  works  on  both  real  and  artificial  data.  W#  are  now  engaged  in  study¬ 
ing  the  algorithm  analytically  and  in  comparing  it  to  both  other  cluster¬ 
ing  techniques  and  to  existing  multivariate  statistical  methods. 

A  convenient  (though  not  usually  well-defined)  division  of  the 
pattern  recognition  problems  Is: 

1)  Design  and  Bvaluation  of  Transducers— here  "transducers" 
are  those  parts  of  a  pattern  recognition  system  that 
transform  a  physical  phenomenon  into  a  set  of  electrical 
measurements  or  optical  patterns  that  are  in  a  form 
suitable  for  the  preprocessor. 

2)  Design,  (Automatic)  Synthesis,  and  Bvaluation  of  Pre¬ 
processing — here  "preprocessing”  is  that  part  of  the 
system  that  transforms  the  measurements  from  the  trans¬ 
ducers  into  multi-dimensional  patterns.  This  transforma¬ 
tion  should  enhance  the  differences  between  classes  we 
want  to  discriminate.  At  the  same  time  it  should  preserve 
wi thin-class  similarities , 
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3)  Design,  (Automatic)  Synthesis,  and  Evaluation  of  the 
Categorizer--here  the  categorizer  is  that  pail  that 
transforms  the  patterns  from  the  preprocessing  into 
labels  (output  codes)  associated  with  each  set  or 
Class  of  nattorns 

In  the  design  of  the  "preprocessing"  to  do  automatic  classification, 
a  first  sufc-task  is  Lo  analyze  a  representative  set  of  patterns.  Many 
present  preprocessing  evaluation  techniques,  even  when  used  on  a  set  of 
representative  data,  give  little  information  as  to  how  to  modify  the 
preprocessing  technique  in  order  to  improve  it. 

In  order  to  know  whether  we  can  Improve  the  preprocessing  we  must 
be  able  to  distinguish  between: 

1)  poor  performance  of  the  preprocessor,  and 

2)  inherently  difficult  data. 

The  proposed  technique  has  demonstrated  its  ability  to  lay  bare  the 
structure  of  the  data.  ISODATA  can  be  used  to  evaluate  preprocessing 
by  comparing  the  clustering  before  the  preprocessing  with  the  clustering 
after  the  preprocessing.  This  makes  it  possible  to  evaluate  the  pre¬ 
processor  with  respect  to  the  inherent  difficulty  of  the  data.  The 
clear  picture  of  the  data  that  the  research er  obtaina  using  this  technique 
helps  him  to  modify  the  preprocessor  so  that  thu  resulting  patterns 
are  structured  in  a  more  suitable  manner.  We  have  found  that  under¬ 
standing  the  structure  of  the  data  is  also  suggestive  of  new  ways  for 
transducing  the  physical  phenomenon. 

In  the  classification  of  patterns  a  primary  problem  la  the  obtaining 
of  an  economical  description  of  the  patterns.  Due  to  the  complex  nature 
of  patterns  arising  from  the  real  world,  a  description  not  rigidly  con¬ 
strained  by  a  number  of  a  priori  assumptions  is  desirable.  The  class 
of  tschnlques  we  describe  has  the  characteristic  that  its  description 
of  the  data  is  determined  primarily  by  the  data  and  is  self-adjusting 
In  a  way  that  makes  this  description  economical. 

In  discussing  this  technique  we  concentrate  on  using  it  for  the 
analysis  of  data.  We  see  the  technique  as  specifically  applicable  to: 

1)  The  design  of  preprocessing  (and  also  more  conventional 
data  analysis) ,  by  allowing  the  examination  of  the 
structure  of  multidimensional  data  in  the  original  high¬ 
dimensional  apace;  and 

2)  Classification  of  patterns,  by  finding  the  structure  in 
each  class  of  patterns  and  providing  an  economical 
description  of  the  classes  of  patterns  against  which  a 
pattern  of  unknown  class  can  be  compared  and  so  assigned 
to  a  specific  class. 
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Ill  THE  TECHNIQ'JE 


* 

ISODATA,  as  the  derivation  of  the  name  suggests,  is  a  collection 
of  iterative  .  It  iUvo  no  i,  auempi  to  summarize  all  of  the 

finest  nuances  extractable  from  the  data.  Rather  it  focuses  on  central 
tendencies  and  the  major  structure  of  the  data.  As  ISODATA  is  normally 
used,  1 l  is  a  compromise  between  attempting  to  store  and  analyze  every 
detail  and  aspect  of  the  data  right  from  the  start  (if  desirable,  this 
can  be  done  later  In  the  analysis  on  limited  portions  of  the  data)  and  an 
approach  that  averages  virtually  everything  together.  (In  fact,  if  It 
were  desirable  for  some  reason,  either  extreme  is  attainable  by  appro¬ 
priate  selection  of  certain  process  parameters  that  control  ISODATA, ) 

It  is  not  practical  to  compare  all  patterns  with  all  other  patterns 
for  large  numbers  of  patterns.  Rather  the  procedure  compares  patterns 
with  a  set  of  clusters  constructed  from  subsets  of  the  patterns  them¬ 
selves,  and  groups  patterns  together  on  the  basis  of  these  comparisons. 

The  comparisons  are  made  by  establishing  a  measure  of  distance  in  the 
measurement  space.  Patterns  are  groupej^together  If  they  lie  closest 
to  the  same  "description  of  a  cluster. "  The  number  of  clusters  used 
by  the  technique  varies  in  a  way  that  depends  on  the  structure  of  the 
patterns  In  measurement  space  and  on  the  ISODATA  process  parameters  that 
the  researcher  controls. 

When  used  on  data  for  which  categorization  information  is  not  avail¬ 
able,  ISODATA  finds  a  good  approximation  to  the  natural  structure  of  the 
data,  rather  than  trying  to  impose  an  assumed  structure  on  the  data.  By 
clustering  only  one  class  of  patterns  at  a  time,  categorization  informa¬ 
tion  can  be  used  in  conjunction  with  ISODATA  to  structure  the  data  for  a 
specific  pattern  classification  problem;  a  probability  distribution  of 
the  data  need  not  be  known  or  even  assumed  to  exist.  The  development  of 
a  computationally- simple  method  that  could  be  Implemented  for  patterns 
of  more  than  ICO  dimensions  (e.g.,  optical  patterns  and  complex  waveforms) 
was  an  important  factor  guiding  the  development  of  this  technique. 

The  simplest  type  of  ISODATA  is  ISODATA-POINTS.  ThiB  technique  is 
described  ir  the  next  section.  In  Appendix  A  we  discuss  ISODATA-LINSS  and 
ISODATA- PLANES,  two  generalizations  of  ISODATA-POINTS .  There  we  give  a 
tentative  algorithm  for  ISODATA- LINES.  Though  neither  of  these  general¬ 
izations  has  yet  been  programmed,  we  feel  them  to  be  relatively  straight¬ 
forward  generalizations  of  ISODATA-POINTS. 


With  apologies  for  adding  another  acronym  to  the  growing  list,  we  have 
coined  ISODATA  to  represent  Iterative  Self-Organizing  Data  Analysis 
Techniques  A.  (The  "A"  was  added  to  make  ISODATA  pronouncabTe . )  The 
classically-oriented  can  derive  it  from  ISO,  meaning  "the  same"  or  "like," 
+  Data.) 

Here  we  use  cluster  in  a  general  way — allowing  it  to  mean  a  set  of 
patterns  grouped  around  a  point,  a  line,  or  a  plane.  Hence  the  "descrip¬ 
tion"  is  the  specification  of  the  point,  line,  or  plane  around  which  the 
patterns  are  clustered. 
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IV  DETAIL£D  DESCRIPTION  OF  ISODATA-POINTS 


Ir  this  section  we  describe  ISODATA-POINTS  from  tour  points  of 
view,  each  succeeding  point  of  view  being  more  precise  than  the  last. 

The  four  points  of  view  are: 

(1)  Verbal 

(2)  Pictorial 

(3)  Two-dimensional  Illustrative  Example 

(4)  Mathematical 

We  also  give  the  results  of  a  principal  components  analysis  (a  more 
conventional  statistical  analysis  technique)  on  the  same  data  so  that 
the  results  of  it  can  be  compared  with  the  type  of  results  obtained 
using  ISODATA-POINTS. 

A.  Verbal  Description 

ISODATA-POINTS  is  an  Iterative  procedure  for  the  sorting  of 
a  set  of  multi-dimensional  (multi-variable)  patterns  into  subsets  of 
patterns.  An  average  pattern  is  used  to  represent  each  subset  of 
patterns,  and  the  iterative  process,  by  changing  ths  composition  of 
these  subsets,  creates  new  average  patterns.  These  new  average  patterns 
define  new  subsets  each  of  which  has  reduced  variation  about  the  average 
pattern.  The  process  also  combines  average  patterns  that  ere  ao  similar 
that  their  being  separate  falls  to  provide  a  significant  amount  of 
additional  information  about  the  structure  of  the  petterns. 

B.  Pictorial  Plow  Diagram 

We  show  a  pictorial  flow  diagram  of  ISODATA-POINTS  in  Fig.  1. 
In  line  with  our  considering  ISODATA  ss  s  procedure  for  sorting  patterns 
we  show  the  patterns  being  fed  Into  a  sorter,  on#  at  a  tlma,  from  a 
"pattern  hopper.  "  The  patterns  are  sorted  into  subsets  on  the  basis  of 
distance  from  a  set  of  cluster  points — esoh  pattern  going  into  that  sub¬ 
set  associated  with  the  cluster  point  to  which  it,  the  pattern,  Is 
closest.  The  cluster  points  themselves  are  obtained  as  an  output  of 
the  previous  iteration.  The  set  of  cluster  points  for  ths  first  itera¬ 
tion  must  be  provided  by  the  researcher. 


The  selection  can  be  arbitrary,  since  the  results  of  clustering  have 
been  found  experimentally  to  be  nearly  independent  of  the  choice  of  the 
initial  cluster  points,  Usually,  however,  a  wise  choice  reduces  sig¬ 
nificantly  the  number  of  iterations  needed  for  satisfactory  clustering. 
We  have  found  it  best  to  use  a  subset  of  the  patterns  randomly  selected 
from  the  training  patterns  ss  the  initial  cluster  points. 
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FIG.  1  A  PICTORIAL  DESCRIPTION  OF  ISODATA- POINTS 


After  all  patterns  have  been  sorted,  the  average  of  each  of  the 
subsets  of  patterns  Is  computed,  and  the  sample  standard  deviations  in 
each  dimension  of  each  subset  are  determined.  The  average  pattern 
vector,  the  standard  deviation  in  each  dimension  for  each  subset,  and 
the  number  of  patterns  in  each  subset  are  then  passed  on  into  the 
” Cluster  inrormation  Hopper." 

Those  small  clusters  (with  fewer  than  ■S  elements)  are  uieoarded  at 
"Valve  1."  The  positioning  of  "Valve  2"  is  determined  by  the  number  of 
the  Iteration  and  by  the  total  number  of  clusters,  as  Indicated  on  the 
diagram. 

The  criteria  and  method  of  splitting  and  lumping  of  clusters  are 
given  in  detail  in  the  next  two  sections.  Splitting  takes  place  if  the 
standard  deviation  in  any  dimension  is  greater  than  0  and  also  if  both 
(1)  the  cluster  has  enough  members  to  split  and  (2)  has  high  average 
distance  between  its  mean  and  the  patterns  in  its  aubsot. 

Lumping  occurs  between,  at  most,  the  L  pairs  of  means  that  are  less 
than  9  apart.  (The  process  parameters  9  ,  9  ,  9  ,  and  L  ara  all  auppllad 
by  thecresecrcher. )  B  c 

After  each  lumping  or  splitting,  the  modified  aet  of  average  point* 
is  used  as  the  set  of  cluster  pointe  for  the  next  iteration  and  placed 
in  the  "Sorter  Box."  The  program  ends  when  the  number  of  lteretiona 
performed  equals  the  number  specified  by  the  researcher.  At  this  stage, 
the  cluster  points  should  adequately  "fit"  the  data. 

C.  Two-Dimension  Illustrative  Example 

In  order  to  illustrate  the  detail*  of  I SO DATA -POINTS  w*  have 
Contrived  the  aet  of  two-dimensional  patterns  shown  In  Tig.  2.  The  two 
dimensions  are  height  and  weight.  The  patterns  (the  points  shown  In 
Fig.  2)  are  intended  to  represent  the  height  and  weight  of  typical 
professional  athletes. 

Given  these  pointe  to  cluster,  the  ISODATA- POINTS  technique 
proceeds  in  a  manner  that  we  illustrate  in  Figs.  2-1B.  Each  figure 
illustrates  a  major  step  in  the  computer  program.  (The  actual  figures 
are  placed  after  the  explanatory  text  for  all  of  the  figures  and  can 
be  folded  out.) 

The  clustering  shown  in  thla  example  was  found  by  running  the 
existing  ISODATA- POINTS  computer  program.  On#  particularly  intereating 
aspect  of  this  run  is  the  way  in  which  the  technique  found  and  isolated 
a  number  of  points  lying  virtually  alone.  This  offers  one  approach 
to  treating  "wild  shots"  in  the  analysis  of  data,  since  they  are 
singled  out  for  further  study  or  for  discarding. 
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Selection  of  the  Pattern  Set 


Note  that  three  distinct  clusters  are  labelled  "Rugby" 

a  nr!  '"belle  -  "ECCltCtbill ,  "  TLsisiuic,  im 

this  problem  a  simple  average,  for  example  of  all 
basketball  players,  will  not  describe  satisfactorily 
a  representative  basketball  player.*  In  other  words, 
the  classes  are  composed  of  several  subclasses,  i.e., 
they  are  "multi-modal." 

Obviously  this  particular  data  set  can  be  analyzed 
with  pencil  and  paper.  A  computer  is  not  necessary 
because  the  data  points  are  described  by  only  two 
characteristics  of  the  athletes,  i.e.,  their  weight 
and  height.  If,  however,  these  data  were  described 
in  many  more  dimensions,  e.g.,  more  than  3  or  4,  then 
it  becomes  nearly  impossible  to  display  them  satisfac¬ 
torily  in  their  raw  form  even  in  many  sets  of  two- 
dimensional  representations. 

We  will  show  in  Section  VI  how  to  obtain  for  these 
cases  a  two-dimensional  plot  that  usefully  describes 
the  data. 

One  important  goal  of  this  ISODATA  analysis  is  a  com¬ 
prehensible  and  useful  description  of  the  data..  In 
order  to  obtain  this  description  we  seek  to  divide 
the  data  points  into  relatively  homogeneous  subsets, 
each  subset  of  which  can  be  adequately  desorlbed  by 
its  average  point.  The  following  example  seeks  to 
describe  bow  relatively  homogeneous  subsets  of  data 
can  be  obtained. 


3 


Selection  of  Initial  Cluster  Points 

Note  that  one  small  region  has  two  initial  cluster 
points  and  another  small  region  has  three.  These 
initial  trial  duster  points  were  selected  to  show 
that  if  by  arbitrary  selection  a  bad  choloe  of  initial 
cluster  points  is  made,  that  even  then  the  final  cluster 
points  will  be  good  ones. 


Partitioning  of  the  Pattern  Space  as  Defined 
Implicitly  by  the  Cluster  Points 


Note  that  the  boundaries  are  the  perpendicular  bi¬ 
sectors  of  the  lines  joining  pairs  of  cluster  points. 
Since  we  are  seeking  minimum  distance  of  a  pattern 
over  all  cluster  points,  the  boundaries  are  meaning¬ 
less  except  where  they  are  between  the  two  closest 

*  Or  to  say  this  in  another  way,  a  man  with  his  head  in  an  oven  and  bis 
feet  on  a  cake  of  Ice  can  hardly  be  adequately  described  as  being 
"warm  on  the  average. 
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Figure 


Step 

cluster  points.  Hence  the  piecewise-linear  nature 
of  the  boundary. 

^  Sorting  vf  .  H  k  b  Ol  Its  i  wl  1  bbl  OilUII  A  UB1  ll|{  tilt? 

Initial  Partition 


Note  that  the  patterns  are  assigned  to  only  one  sub¬ 
set,  and  that  all  subsets  are  contained  In  convex 
volumes  of  pattern  space.  Note  also  that  the  Initial 
partition  Is  not  a  good  one.  Subsets  having  fewer 
than  8  elements  would  be  discarded  at  this  point. 

(8  lsNa  researcher-supplied  process  parameter.) 

N 

It  may  be  helpful  in  this  particular  example  to  think 
of  the  data  points  as  representing  men  standing  in  a 
large  field.  The  man  are  positioned  In  the  field  In 
accordance  with  their  weight  and  their  height.  The 
partitions  that  divide  the  data  points  Into  sub¬ 
sets  can  be  thought  of  as  "fences"  dividing  the  men 
into  groups.  The  cluster  points  can  be  considered 
as  "group  leaders"  to  whom  the  men  owe  temporary 
allegiance,  i. e. ,  a  man  owes  his  allegiance  to  the 
closest  group  leader.  As  we  shall  sea,  In  the  XBODATA 
process  group  leaders  coma  and  go  (it  was  ever  thus) . 

0  Finding  the  Average  Point  of  Each  Subset 

After  the  first  Iteration  the  ISODATA  average  points 
become  cluster  points. 

7  Splitting  of  the  Average  Points  takes  place  when 

(1)  the  maximum  standard  deviation  exceeds  8  (a 
researoher-aupplled  ISODATA  process  parameter) 
and  either  a  or  3  is  true. 

(2)  the  number  of  patterns  in  a  subset  exceeds 
(28  +  2)  and  (when  the  average  distance  of 
patterns  in  subset  1  from  the  average  point 
of  subset  1)  exceeds  AD,  the  average  distance 
of  a  pattern  from  its  closest  average  point. 
NROWS  is  tha  number  of  oluaters 


*  A  volume  la  convex  if  the  straight  line  connecting  any  two  points  in 
the  volume  lies  entirely  within  the  volume. 
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Figure  Step 

7  (Cont.d)  More  precisely, 

nnOWS 

AD  *  1=1  (AVEDST^  x  (N^) 


(Distance  of  pattern  J  from 
J-l  the  mean  vector  of  subset  1) 

(For  all  patterns 
in  subset  1) 

(3)  The  number  of  oluaters  is  leas  than  or  equal 
to  one-half  the  number  of  clusters  that  the 
researcher  has  specified  as  being  desired. 

The  Splitting  occurs  in  the  following  fashion: 

If  the  conditions  of  splitting  are  satisfied 
then  the  first  new  average  points  (for  example, 
the  right  hand  or  the  upper  average  points 
of  Fig.  7)  are  created  by  adding  1  to  that 
component  of  the  original  average  point  having 
the  largest  standard  deviation.  The  second 
"average  points"  (for  example,  the  left  hand 
or  the  lower  average  points  of  Fig.  7)  are 
created  by  subtracting  1  from  that  component 
of  the  original  average  point  having  largest 
standard  deviation. 


and 

AVEDST  =  ^ 
1  " 


The  actual  amount  added  is  arbitrary  (here  it  is  +1)  so  long  as  it  is 
sufficient  to  provide  detectable  differences  in  the  distance  of  a 
pattern  from  the  two  cluster  centers  and  Is  not  so  large  as  to  change 
other  boundaries  appreciably. 


8. 


The  Partition  for  Iteration  2. 


Note  that  the  boundaries  between  the  pairs  of  cluster 
points  split  from  a  single  point  are  perpendicular  to 
the  direction  having  maximum  standard  deviation. 

t  Sorting  of  the  Patterns  for  Iteration  2. 

10.  Finding  the  Average  Points  of  Iteration  2. 

Note  the  effect  of  the  "outlier"  (shown  with  an  arrow) 
on  the  average  point  for  the  uppermost  cluster. 

11 •  The  Average  Points  found  In  Iteration  3. 

In  iteration  2  the  average  points  were  again  split 
(since  the  number  of  subsets  was  less  than  one-half 
the  number  of  subsets  desired) .  Note  that  the 
"outlier"  of  Fig,  10  has  been  made  a  cluster  by  itself. 

12.  The  Average  Points  of  Iteration  3  are  Split  In  the 
banner  Described  under  Fig,  7  above. 

13.  The  Sorting  of  the  Patterns  in  Iteration  4. 

14.  The  Finding  of  Average  Points  for  Iteration  4. 

18.  The  Lumping  Together  of  Close  Average  Points. 

In  this  iteration  the  criteria  for  lumping  (an  even 
Iteration  and  the  existence  of  more  than  one-half  the 
number  of  subsets  desired)  are  satisfied.  This  figure 
illustrates  the  lumping  together  of  all  pairs  of 
average  points  that  are  leas  than  a  dlstanoe  of  Ac 
apart.  ( is  a  researcher- supplied  ISODATA  process 
parameterr.  Note  th|J  only  pairs  of  average  points 
are  lumped  together.  Note  also  that  the  lumped 
average  point  obtained  in  the  average  of  the  two 
average  points  and  is  obtained  by  weighting  each 
average  point  by  the  number  of  patterns  in  its  sub¬ 
set.  This  makes  the  lumped  average  point  the  true 
average  point  of  the  combined  subset. 

* 

Splitting  of  average  points  in  several  dimensions  (into  more  than  two 
new  "average  points")  was  once  considered  for  use  in  the  algorithm. 

We  found  that  this  becomes  hazardous  unless  the  covariance  matrix  is 
calculated,  and  this  calculation  la  undesirable, 
we 

Lumping  triples  was  considered  for  the  algorithm,  and  discarded,  since 
it  appeared  to  change  the  partition  too  radically  for  the  iterative 
procedure  to  satisfactorily  "converge." 


16. 


The  Partition  for  Iteration  5. 
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19. 


Tho  iifO»*Q<TO  nf  o  ■+■  Vi  r»  OuVi  pn+p  T  +  4  r\**  R 


The  Average  Points  for  Iteration  6, 

In  the  previous  iteration,  six  average  points  ware 
split.  In  this  Iteration  four  pairs  of  average  points 
will  be  lumped  together.  These  four  pairs  of  points 
are  indicated  by  being  circled. 

The  Final  Average  Points  After  Several  Iterations. 

No  splitting  or  lumping  is  allowed  in  this  final 
Iteration,  which  is  principally  for  consolidation. 

We  feel  that  these  19  average  points  do  quite  ade¬ 
quately  describe  thiB  set  of  562  data  points  with 
a  30:1  reduction  of  the  number  of  points.  Naturally, 
no  description  of  the  original  data  points  which 
provides  a  similar  amount  of  reduction  can  be  as 
accurate  as  the  original  data  itself.  However,  a 
reduced  data  description  is  often  more  useful  than 
the  more  accurate  but  much  more  voluminous  version. 

Note  the  way  the  "wild  shots"  or  "outliers"  have 
been  found  and  isolated  by  assigning  them  to  clusters 
of  their  own.  These  wild  shots  csn  now  be  examined 
for  their  Importance— either  as  a  rare  occurrence 
well  worth  notlng,or  as  an  equipment  malfunction. 

We  terminated  the  iterative  procedure  after  itera¬ 
tion  7,  because  the  clustering  obtained  seemed  quite 
adequate.  If  it  had  been  necessary,  we  could  have 
gone  on  by  continuing  the  lumping  and  splitting, 
starting  at  the  end  of  iteration  6.  Our  experience 
has  shown  that  abgut  six  iterations  are  adequate 
for  many  problems  --adequate  in  the  sense  that  the 
number  of  clusters  is  stable  and  the  subsets  rela¬ 
tively  homogeneous, 


We  can,  by  increasing  9  and  increasing  8  reduce  the 
number  of  clusters  we  oStain.  Decreaslngsthem  both 
would  increase  the  number  of  clusters. 

The  number  by  each  average  point  gives  the  number  of 
patterns  in  that  cluster. 

* 

We  are  presently  seeking  better  criteria  for  terminating  the  iterations. 


FIG.  6  FINDING  THE  AVERAGE  POINT  OF  EACH  SUBSET 


WEIGHT  — LBS 


56  63  68  73  76  83  88 


HEIGHT —  INCHES 


FIG.  7  SPLITTING  OF  THE  AVERAGE  POINTS 


AVERAGE  POINTS  NOT 


63  68  73  78  83 

HEIGHT  — INCHES 

FIG.  15  THE  LUMPING  TOGETHER  OF  CLOSE  AVERAGE  POINTS 
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Mathematical  Description 

The  details  o{  the  calculations  made  In  the  existing  ISODATA- 
avima  computer  program  are  given  in  this  section. 

In  Pig.  20  we  show  a  computational  flow  chart,  of  the  technique 
A  glossary  of  the  symbols  used  in  the  mathematical  description  is  given 
next  in  order  to  ease  the  struggle  with  new  notation.  Following  the 
glossary  we  explicitly  write  the  mathematical  expression  In  the  sequence 
calculated  for  each  significant  computation  made  by  the  program. 

Readers  not  interested  in  the  details  of  the  computation  made 
by  ISODATA-POINTS  can  skip  this  section  (i.e.  turn  to  page  42)  with  out 
serious  lose. 


* 

The  processing  time  for  this  program  is  about 
r  -4 

12.7  x  10  x  (number  of  patterns)  x  (number  of  cluster  points) 
x  (number  of  dimensions)  ]  seconds/iteration 
on  the  B-5500  computer  (at  $1 80/hr)  at  SRI.  The  program  is 
written  in  Algol  80. 
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START 


FIG.  20  A  FLOW  DIAGRAM  SHOWING  THE  COMPUTATIONAL  CYCLE 
OF  ISODATA-POINTS 
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SYMBOLS  USED  IN  MATHEMATICAL  DESCRIPTION 


SYMBOL 

STANDS  FOR 

FOUND  IN 
STEP  NOS. 

AD 

Overall  average  distance  of 
patterns  from  the  average  vector 
of  the  cluster  to  which  they  are 
assigned.  NRflta 

AD  =  -i  /  (AVBDST. )  x  N. 

N  4  1  1 

6,  10 

AVBD8T1 

The  average  distance  of  the  patterns 
in  cluster  C  from  the  average 
vector  (average  point)  of  that  cluster 

V 

*™"i  ‘Tt  V 

for  all  P  eC 

J  1 

4,  6,  10 

Ci 

The  1th  cluster 

D 

The  dimension  (number  of  components) 
of  a  pattern  vector. 

5,  9 

6U 

The  Euclidean  distance  between  the 
average  vector  P  for  cluster  C 
and  the  averagevector  P  for 
cluster  Cj  ^ 

-V  (fP 

12 

L 

The  maximum  number  of  pairs  of 
average  vectors  that  can  be 
lumped  at  one  time. 

12,  13 

"i 

T|}|>  cluster  point  (vector)  for  the 
il  cluster 

m 

SYMBOL 


STANDS  FOR 


The  total  number  of  patterns. 


The  number  of  pattern*  in  the 
1  cluster  . 


The  total  number  of  clusters. 
(Stands  for  the  Number  of  Rows 
In  the  matrix  having  the  cluster 
point  vectors  as  rows,  which  la 
an  NRWS  x  D  matrix.) 


The  J  Pattern  vector. 


The  X*  component  of  the  J4 
pattern  vector  P^ . 


The  averagg  pattern  vector 
for  the  lxn  cluster 


The  Xth  component  of  the  average 
pattern  vector  P  for  the  i* 
cluster. 


The  positively  '[spilt"  part  of  the 
average  vector  ^P.  (See  Step  10.) 


The  negatively  'split”  part  of  the 
average  vector  ^P.  (See  Step  10.) 

The  standard  devlat^gn  of  the  1th 
cluster  C,  In  the  i  component 
(dimension) _ __ 


POUND  IN 
STEP  NOS. 


3  a  fi 

»  —  I  C  I 

10,  13 


2,  4,  5, 

6,  7,  9, 
10,  11,  IS 
14 


3,  4,  10, 
12,  13 


ar ^ 


‘  f^^ry^Piwwwg^ffrwt  *?  »m,T--t, toss lraiM^gaMjaaPtWqrjl 


SYMBOL 

STANDS  FOR 

FOUND  IN 
STEP  NOS. 

iCk 

The  largest  standard  deviation 
of  all  of  the  component*  of  the 
patterns  in  cluster  ,  The 

largest  |£andard  deviation  occurs 
in  the  kx  component. 

9,  10 

i°k  *  T  fij } 

ec 

The  ISODATA  process  parameter  against 
which  the  distance  &  between  pairs 
of  patterns  is  compared.  It  controls 
the  "lumping"  process.  It  is  supplied 
by  the  researcher. 

12 

9k 

The  ISODATA  process  parameter  against 
which  the  maximum  standard  deviation 
jPk  is  compared.  It  controls  the 
splitting  process.  It  is  supplied 
by  the  researcher. 

10 

9N 

The  ISODATA  process  parameter  against 
which  the  number  N  of  patterns  in  a 
cluster  is  compared.  It  is  supplied 
by  the  researcher. 

7,  10 
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STATEMENT  OF  THE  GOAL 


Given  a  set  of  pattern  vnotnr*  [p  j  =  i,  vf  uimonaxon  u, 

the  goal  of  ISODATA- POINTS  Is  to  sort  them  into  subseta  C  each  having 
N^  members  and  having  small  withln-group  variance,  l.e.,  find  a  sot  of 
average  vectors  such  that  each  of  these  average  vectors  adequately 
describes  that  set  of  patterns  lying  closest  to  it.  (Measures  and 
criteria  for  determining  when  an  adequate  description  of  the  data  has 
been  obtained  are  being  sought.) 

The  following  steps  describe  in  symbols  the  manipulations  shown 
graphically  in  the  figures  accompanying  the  two-dimensional  example 


illustrating 

ISODATA-POINTS. 

Step 

Computation 

1. 

Select  arbitrary  subset  of  patterns  having  NROWS 
elements.  These  should  be  chosen  as  intelligently 
as  posslble>-i .e. ,  if  possible,  one  from  each 
known  sub-class  or  cluster.  Create  a  set  of 

points  that  are  duplicates  of  this  randomly 

selected  set  of  patterns.  Call  this  duplicate 
set  the  initial  "cluster  points,"  ■  {M,  ,  1=1,  — , 

nrows}  1 

2. 

Do  for  all  >1, - ,M:# 

For  each  pattern  P^  find  such  that 

(Pj-  M1*)-(PJ-M1*)  -  M^n  [(Pj-M1)*(PJ-Mi)] 

where  the  dot  product 

A’B  =  <at . • (b1 . . . ,bD) 

D 

"  E  \bi 

iul  1  1 

Assign  P  to  subset  C  +.  This  step  sorts  the  {P4} 
j  l  J 

into  subsets  on  the  basis  of  distance  from  the 
[Mi)  (see  figure  below).  (Ties  are  arbitrarily 
decided.  They  almost  never  occur, )„ 

/Vrv 

0 

Note  that  the  M  are  not  changed  during  this  calculation  over  all  J. 
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Compute  for  all  of  the  1  clusters  C  the  average 
vector  of  each  cluster  i 

,p  =  IT  2  *V 

1  pj6Ci  J 

where  is  the  number  of  patterns  in  cluster 


The  £  P  is  obtained  as  patterns  are  being  sorted. 


For  all  i ,  1=1, ,  NROWS. 


Compute  the  average  distance  AVSD6T,  of  patterns 
in  cluster  from  the  average  vector,  of  that 
cluster. 


av,d8ti  -  ft  iii  yVi*) ’ (pap) 


For  all  P^cC^ 


For  all  i,  1-1, ... ,NR0WS,  and  for  all  i,  ^1,.  .  ,  ,D 
find  the  standard  deviation  o  .  of  the  i  subset 
for  £  measurement  where 


°i£  " 


r  i  —  2 

jil  CpjX"iP/) 


For  all  PjCC^ 


Compute  average  distance  overall,  AD,  where 


__  NROWS 

AD  =  ~  £  (AVEDST  )  x  N, 

N  i„l  1  1 


This  Is  the  average  distance  of  the  patterns  from 
their  closest  cluster  point. 


Computation 


Step 

NOTE: 


Certain  parameters,  6  ,  ©  ,  0  ,  L  and  the  total  number  of 
Iterations,  which  will  be  mentioned  in  the  following 
steps,  are  provided  the  program  bv  the  reswwher 


For  Steps  7-10  and  Steps  12-13  no  use  Is  made  of  the 
individual  patterns.  All  calculations  are  made  based  on 
means,  standard  deviations,  AVEDST^,  and  AD,  and  the  process 
parameters . 


7.  For  all  1,  1=1,...,  NBOWS. 

If  N  <  0  then  discard  the  ith  cluster,  C  ,  and  reduce  the 
number  or  clusters  by  1. 


a.  - 

(a) 

If  this  Iteration  is  the  last  iteration,  set  ©  =  0  and 
skip  to  step  12. 

(b) 

If  number  of  clusters  Is  less  than  or  equal  to  $  the 
number  desired,  then  skip  the.  remaining  step  in  8  and 
do  step  0. 

•  ■ . 

(c) 

If  this  iteration  le  an  even- numbered  iteration,  or 
if  number  of  clusters  is  greater  than  or  equal  to 
twice  the  number  desired,  then  skip  to  step  12. 

NOTE: 

Steps  0  through  11  comprise  the  so-called  "splitting" 
process. 

9. 

For 

all  i,  i=l, NROWg,  for  J=1 , . . . ,D. ,  find  k  and 

i°k 

such  that 

i^k  =  m!x(a1J) 

10. 

For 

each  i,  1=1,. . . , NROWS 

(a) 

If  iffk  >0_  and 
c» 

((if  AVEDSTt  >AD  and  if  Nt  >2®„+  2)  or 
(NROWS *0.5  x  number  of  clusters  desired)) 


then  create 
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Step 


Computation 


10 . (Cont| ) 


11. 


NOTE: 


12. 


,P+  =  ,P  +  (0 . 0.+1.0 _ O) 

^ — . — ^~~-(ktncomponent) 

The  +1  is  placed  in  this  k  component 
element  (corresponding  to  for  duet er  i) 
in  order  to  split  the  cluster  along  the 
direction  having  the  maximum  variation. 

p-D  P  +  (0....0  ,-l,...,0) 

1  H  — - (k  n  element) 

(b)  P  is  replaced  by  ^P  +,  and  is  added  to 

the  list  of  average  vectors,  which  increases 
the  number  of  clusters  by  1.  (See  figure  below) 


Start  the  process  again  at  Step  2.  Use  the 
f  P,  lal | . . . ,NR0WS ' }  as  the  new  set  {Ml] in 
place  of  the  existing  set  {M.}.  (NROWSMs 
the  number  of  clusters  af ter  splitting  or 
lumping.) 


Steps  12  through  14  comprise  the  so-called  "lumping" 
process. 

For  all  i,  ini,...,  NROWS: 

For  all  J  >  1,  j-i  +  1, . .  ..NRWSi 

(1)  Compute  the  pairwise  distance  5  between 
average  points  where  J 


B 


U 


P 


... 


Step 


Computation 


12.(Cont.)  (2) 


If  5  <  9C  then  place  6  ,  i£,  in  an 

ordered  (3  x  L)  matrix. 
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L  (which  Is  £  9  for  programming  convenience)  controls 
the  maximum  number  of  pairs  of  clusters  that  are 
lumped  together. 


For  all  l,  1=1,, 


L. 


If  P  and  P  have  not  been  previously  used  In 
V,  3i 
lumping,  then 

(1)  Compute 

XL  V 


+  N. 


(2)  Replace  P  with  P*  and  delete  P  from  the 
Xl  1jt 

Hat  of  average  vectors  (reducing  the  number  of 
cluatera  by  1) . 


y 


14,  If  more  Iterations  are  to  be  done  (this  is  at  dis¬ 

cretion  of  investigator) ,  start  the  process  again 
at  Step  2.  Use  the  {  P,  1=1, . . .NROWS ' }  as  the  new 
set  [M'}  in  place  of  the  existing  set  fl^}.  If  this 
_ was  the  last  Iteration,  then  end  process. _ 
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E.  Analysis  of  the  Height  vs.  Weight  Data  Using  Principal 
Components  Analysis 


An  alternative  method  of  describing  and  analyzing  the  data  of 
Section  C  (the  two-dimensional  example)  would  be  principal  components 

o n J 


It  may  be  objected  that  principal  components  analysis  should  not 
be  applied  to  data  that  is  as  heterogeneous  as  the  data  in  this  two-dimen¬ 
sional  example.  We  agree.  At  least  part  of  our  point  is  that  it  is  not 
easy  in  high  dimensions  to  determine  just  how  heterogeneous  the  data  is. 

"The  (Principal  Component  Method)  is  a  relatively  straight¬ 
forward  way  of  ’breaking  down'  a  covariance  or  correlation  matrix  into 
a  set  of  orthogonal  components  or  axes  equal  in  number  to  the  number  of 
variates  concerned.  These  correspond  to  the  latent  roots  and  the  accom¬ 
panying  latent  vectors, .. .of  the  matrix.  The  method  has  the  property  that 
the  roots  are  extracted  in  descending  order  of  magnitude,  which  is  important 
if  only  a  few  of  the  components  are  to  be  used  in  summarizing  the  data. 

The  vectors  are  mutually  orthogonal ,  and  the  components  derived  from  them 
are  uncorrelated."  The  greatest  possible  "scatter"  of  n  points  pro¬ 
jected  onto  a  given  number  a  of  coordinate  axes  in  a  k- dimensional  space 
(s  *  k>  is  obtained  by  this  method. 

The  average  point  of  the  height  vs.  weight  data  is  (724,208), 

(the  inches  are  multiplied  by  10)  and  the  covariance  matrix  is 

2667  1111 

1111  2814 

The  first  eigenvalue  is  3854  and  the  corresponding  eigenvector 
is  (.936,1.00).  The  second  eigenvalue  is  1627  and  the  corresponding  eigen¬ 
vector  is  (1.00,  -.936).  In  Fig.  21  we  have  plotted  these  eigenvectors 
as  a  second  set  of  "coordinate  axes"  with  the  mean  value  of  all  of  the  data 
as  origin.  The  length  of  the  vectors  is  proportional  to  the  magnitude  of 
the  associated  eigenvalue. 

The  direction  of  the  first  eigenvector  indicates  tbat  generally 
there  is  a  positive  correlation  between  height  and  weight;  that  is,  weight 
Increases  with  height.  This  "accounts  for"  about  70%  of  the  variance. 

Both  height  and  weight  contributed  about  equally  to  this  component. 

The  second  eigenvector  displays  the  extent  to  which  height  and 
weight  are  negatively  correlated.  Again  both  height  and  weight  contribute 
about  equally  to  this  component. 


$ 

The  exact, values  of  "scatter"  for  n  data  points  in  k  dimensions  are  given 
by  Wilks. 
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These  descriptions  relate  primarily  to  directions  in  the  data. 

The  largest  eigenvector  gives  the  direction  along  which  a  scale  should  be 
set  up  to  get  maximum  variation  in  the  data  r*  the  goal  v.f  the  dam 
analysis  is  to  find  such  a  scale,  then  this  is  a  very  reasonable  analysis 
technique . 

However,  such  "direction-finding"  techniques  tend  to  ignore 
details  in  the  data  such  as  the  existence  of  Isolated  clusters,  e.g.,  the 
cluster  around  the  point  (32",  115  lbs.)  in  Fig.  21. 

Clustering  techniques  like  ISODATA-POINTS  ignore  direction  while 
clustering  the  data  in  the  pattern  space.  However,  the  average  points 
obtained  can  be  used  to  derive  the  directional  characteristics  of  the 
data.  They  are  primarily  sensitive  to  the  density  of  the  patterns  in 
pattern  space.  They  are  well  suited  to  "zooming"  in  on  the  detailed 
structure  of  the  data.  They  also  can  serve  as  methods  for  a  preliminary 
sorting  of  patterns  into  relatively  homogeneous  subsets  for  further 
statistical  processing.  (This  sorting  con  prevent  the  confounding  of  two 
disparate  effects  resulting  from  treating  these  effects  as  if  they  were 
the  result  of  the  same  (simple)  cause.) 

It  may  also  be  objected  that  1S0DATA-P0INTS  has  a  certain  arbi¬ 
trariness  about  it  and  that  by  setting  the  process  parameters  differently 
we  would  obtain  different  average  points.  It  is  true  that  different 
average  points  can  be  obtained  by  varying  the  process  parsmeter*.  How¬ 
ever  the  results  of  the  clustering  plus  specification  of  the  cluster 
parameters  used,  provides  an  obj active  and  useful  description  of  the 
data.  ” 

In  complex  data  we  have  found  that  there  are  a  variety  of  valid 
clusterings  depending  on  the  number  of  average  paints  used,  on  scaling, 
and  on  the  structure  of  the  data  itself.  For  example,  if  the  data  con¬ 
sists  of  tight  clusters  of  data  whose  dlstancee  apart  are  large  with 
respect  to  the  "diameter"  of  a  cluster  then  the  number  of  clusters  will 
not  vary  even  with  wide  variations  in  the  process  parameters.  If,  on 
the  other  hand,  the  data  is  uniformly  distributed  in  pattern  space,  then 
the  number  of  clusters  found  will  tend  to  vary  rather  smoothly  with 
changes  in  the  process  parameters.  The  way  that  the  number  of  clusters 
varies  as  a  function  of  the  process  parameters  can  be  used  to  describe 
the  structure  of  the  data. 

For  these  reasons,  we  feel  that  what  must  seem  arbitrariness  to 
some  is  a  flexibility  that  is  needed  for  the  analysis  of  real  data.  We 
feel  that  this  flexibility  is  not  detrimental  in  the  case  of  data  analysis 
hy  clustering. 

We  agree  with  John  Tukey’s  appeal  for  good  judgment  in  place  of 
rigorous  optimization- 

"Scientists  know  that  they  will  sometimes  be  wrong;  they  try  not 
to  err  too  often,  but  they  accept  some  insecurity  as  the  price 
of  wider  scope.  Data  analysts  must  do  the  same."i6 
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And  further. 


"if  data,  analysis  is  to  be  well  done,  much  of  it 
T.  J t  be  c  .v  tic,  \j2  juu^mciiL ,  aiiu  theory,  wnetner 
statistical  oj?non-statistical ,  will  have  to  guide, 
not  command." 

And  finally,  we  quote  from  Tukey  at  some  length,  because  of  the  rele¬ 
vance  of  his  remarks  to  clustering  techniques, 

"Practicing  data  analysis':  If  data  analysis  is  to 
be  helpful  and  'useful,  it  must  be  practiced.  There 
are  many  ways  In  which  it  can  be  used,  some  good  and 
some  evil.  Laying  aside  unethical  practices,  one  of 
the  most,  dangerous  (as  I  have  argued  elsewhere  (Tukey, 
1961b))  1s  the  use  of  formal  data-analytlcal  procedures 
for  sanctification,  for  the  preservation  of  conclusions 
from  all  criticism,  for  the  granting  of  an  imprimatur. 

While  statisticians  have  contributed  to  this  misuse, 
their  share  has  been  small.  There  in  a  corresponding 
danger  for  data  analysis,  particularly  in  its  statistical 
aspects.  This  is  the  view  that  all  statisticians  should 
treat  a  given  set  of  data  in  the  aame  way,  Just  as  all 
British  admirals,  in  the  days  of  sail,  maneuvered  in 
accord  with  the  same  principles.  The  admirals  could  not 
communicate  with  one  another,  and  a  single  basic  doctrine 
was  essential  to  coordinated  and  effective  action.  Today 
statisticians  can  communicate  with  one  another,  and  have 
more  to  gain  by  using  special  knowledge  (subject  matter 
or  methodological)  and  flexibility  of  attack  than  they 
have  to  lose  by  not  all  behaving  alike, 

in  general,  the  best  account  of  current  statistical 
thinking  and  practice  is  to  be  found  in  the  printed 
discussions  in  the  Journal  of  the  Royal  Statistical 
Society.  While  reviewing  some  of  these  lately,  I  was 
surprised,  and  a  little  shocked  to  find  the  following: 

’I  snould  like  to  give  a  word  of  warning  concerning  the 
approach  to  tests  of  significance  adopted  in  this  paper. 

It  is  very  easy  to  devise  different  tests  which,  on  the 
average,  have  similar  properties,  i.e.,  they  behave  satis¬ 
factorily  when  the  null  hypothesis  is  true  and  bave  approxi¬ 
mately  the  same  power  of  detecting  departures  from  that 
hypothesis.  Two  such  tests  may.  however,  give  very 
different  results  when  applied  to  a  given  set  of  data. 

This  situation  leads  to  a  good  deal  of  contention  amongst 
statisticians  and  much  discredit  of  the  ecience  of 
statistics.  The  appalling  position  can  easily  arise  in 
which  one  can  get  any  answer  one  wants  if  only  one^oes 
around  to  a  large  enough  number  of  statisticians.  ' 
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To  my  mind  tills  quotation,  if  taken  very  much  more 
seriously  than  I  presume  it  to  have  been  meant, 
nearly  typifies  a  picture  of  statistics  as  a  mono¬ 
lithic,  authoritarian  structure  designed  to  produce 

t  hp  '  Af  f  H  n|o1  1  ruailUc  'YlllXC  pCC«lbi  11  tV  C* 

development  in  this  direction  is  a  real  danger  to 
data  analysis,  I  find  it  hard  to  believe  that  this 
danger  is  as  great  as  that  posed  by  over-emphasis 
on  optimization. 

Facing  uncertainty."  The  most  important  maxim  for 
data  analysis  to  heed,  and  one  which  many  statis¬ 
ticians  seem  to  have  shunned,  is  this:  'Far  better 
an  approximate  answer  to  the  right  question,  which 
is  often  vague,  than  an  exact  answer  to  the  wrong 
question,  which  can  always  be  made  precise. '  Data 
analysis  must  progress  by  approximate  answers,  at  best, 
since  its  knowledge  of  what  the  problem  really  is  will 
at  beet  be  approximate.  It  would  be  a  mistake  not  to 
face  up  to  this  fact,  for  by  denying  it,  we  would  deny 
ourselves  the  use  of  a  great  body  of  approximate 
knowledge,  as  well  as  failing  to  maintain  alertness 
to  the  possible  importance  in  each  particular  Instance  .. 
of  particular  ways  in  which  our  knowledge  is  Incomplete . " 

We  are  presently  investigating  In  more  detail  the  relation¬ 
ships  between  clustering  and  "direction- finding"  techniques.  It  appears 
at  this  time  as  though  they  are  qualitatively  different,  and  that  they 
should  be  used  to  complement  each  other  In  data  analysis. 


V.  EXPERIMENTAL  RESULTS  FROM  COMPUTER  IMPLEMENTATION  OF  ISODATA-POINTS 

In  order  to  understand  and  evaluate  the  technique,  we  have  per¬ 
formed  a  series  of  experiments.  These  experiments  have  been  of  three 
types:  those  designed  to  validate  ISODATA-POINTS,  those  designed  to 
illustrate  graphically  the  approach  taken,  and  those  designed  to  ana¬ 
lyze  data  from  the  real  world. 

The  detailed  results  that  we  have  obtained  using  the  ISODATA-POINTS 
program  will  be  contained  in  a  larger  report.  We  shall  not  attempt  to 
repeat  the  contents  of  that  report,  but  rather  shall  extract  some  of 
the  results  that  we  consider  particularly  significant. 

The  validation  experiments  were  constructed  from  data  whoee  struc¬ 
ture  was  well  known.  In  order  that  we  might  evaluate  the  clustering 
obtained  by  ISODATA-POINTS.  The  data  was  obtained  by  adding  Gaussian 
random  noise  to  10  prototype  patterns  which  serve  as  the  ideal  versions 
of  the  noisy  vectors.  Each  vector  had  10  analog  dimensions  and  can  be 
displayed  as  shown  In  Fig.  22(a).  The  values  in  each  dimension  were 
coded  into  a  10-bit  binary  number  using  "snake-in-the-box"  codes 
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(a  modified  gray  scale  coding) .  These  10  dimensions  were  then  com¬ 
bined  to  give  a  100-bit  pattern.  The  closest  intermean  distance  was 
108  units  of  the  original  10-dimensional  space,  or  roughly  20  bits 
aoart  in  Hammins'  distance.  The  onvarl ance  matrix  was  the  same  1 nr  all 
distributions  and  was  the  scalar  matrix,  Cl ,  I  being  the  Identity 
matrix,  and  J?v30  units,  where  a  is  the  standard  deviation  of  the  dis¬ 
tributions.  The  of  u— 30  is  also  indicated  in  Fig.  22(a). 

The  patterns  were  processed  by  the  IS0DATA-P0INT3  program  without 
specifying  the  distributions  from  which  they  came.  The  process  para¬ 
meters  were  varied  by  the  authors  until  the  program  sorted  the  patterns 
into  11  clusters.  We  assigned  each  cluster  to  the  mode  whose  patterns 
predominated  in  that  cluster.  Doing  this  the  program  classified  98% 
of  the  patterns  correctly.  The  Bayesian  decision-theoretic  optimum 
separating  planes,  which  were  positioned  using  a  priori  knowledge  of 
the  location  of  the  means,  achieved  a  99%  correct  rate  using  unquantized 
data. 


In  a  later  experiment  only  two  distributions  were  used.  The  wave¬ 
forms  of  the  two  means  are  shown  in  Fig,  22.  Their  means  were  but  56 
units  (about  10  bits)  apart,  while  they  still  had  a  standard  deviation 
of  about  30  units  (about  6  bits).  We  obtained  estimates  of  the  mean 
values,  again  without  knowledge  of  the  pattern  mode  from  which  the  data 
came.  The  values  obtained  were  only  slightly  different  from  the  correct 
means.  The  optimum  decision  plane  gave  a  percentage  correct  classifica¬ 
tion  of  81%  while  ISODATA- POINTS  (using  two  large  clusters  and  two  quite 
small  oneB)  obtained  78%. 

In  a  second  validation  experiment  we  constructed  48  pattern  vectors, 
half  of  which  had  the  last  six  bits  +  l’s  and  half  of  which  had  the  last 
six  bits  -  l’s.  The  first  24  bits  of  these  30-bit  vectors  were  then 
filled  with  pattern  vectors  positioned  so  as  to  have  all  pairwise  Hamming 
distances  between  patterns  exactly  24  bits  in  this  24-dimensional  space. 
The  ISODATA- POINTS  program  proved  capable  of  extracting  the  six  consistent 
bits  of  this  30-bit  vector — disregarding  thr  rest  of  the  "noise"  for  which 
bits  were  not  consistent.  The  program  was  100%  correct  in  its  classifica¬ 
tion  of  these  vectors.  Again  no  categorization  information  was  used  to 
cluster  the  patterns. 

In  another  experiment  designed  primarily  to  illustrate  ISODATA- POINTS 
graphically,  we  drew  a  set  of  0's  and  a  set  of  Q's  on  a  10  x  10  retina 
of  squares.  The  0's  and  Q’s  had  no  registration  noise  (i.e.,  were  not 
translated  or  rotated)  but  did  have  bits  of  noise  added  randomly,  ad¬ 
jacent  to  the  main  outline  of  the  0's  and  Q's,  The  program  proved  cap¬ 
able  of  dividing  the  0's  from  the  Q's  perfectly.  In  addition,  the  sub¬ 
tracting  of  the  average  "0"  pattern  from  the  average  "Q"  pattern  em¬ 
phasized  the  fact  that  the  tall  of  the  Q's  was  the  primary  distinction 
between  these  two  classes  of  patterns. 
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We  felt  that  the  utility  of  ISODATA- POINTS  would  be  most  clearly 
indicated  by  application  to  data  drawn  from  actual  research  situations. 
We  chose  two  such  situations.  The  first  was  the  sorting  of  a  9et  of 
sociological  questionnaires,  relating  to  the  attitude  of  209  scientists 
at  t/aftmta  d  i  r  Pfirpe  l  ohrtr*fl  t  rtri  Afl  .  Tn  this  instance  it  was  somewhat 
difficult  to  specify  a  reasonable  sorting  of  the  questionnaires  into 
groups  or  categories.  The  second  was  a  set  of  weather  data,  relating 
to  ceiling  height  prediction  in  Washington,  D.C.  In  this  situation, 
we  could  perform  a  preliminary  sort  (i.e.,  define  a  classification) 
using  the  ceiling  height  that  actually  did  occur. 

The  sociological  data  were  obtained  from  the  Systems  Analysis 
Laboratory  of  the  Management  Sciences  Division  of  Stanford  Research 
Institute.*  We  found  the  groups  into  which  the  ISODATA- POINTS  technique 
divided  the  scientists'  questionnaires  had  reasonable  internal  con¬ 
sistency  as  measured  by  the  mean  deviation  from  the  average  point. 
Conversations  with  the  research  sociologists  have  indicated  that  these 
groupings  are  meaningful  in  terms  of  their  experience  with  the  personnel 
in  the  laboratories.  We  were  even  able  to  obtain  distances (in  terms 
of  the  measurements  made  by  the  questionnaire)  between  the  average 
points  of  these  groups  and  to  obtain  the  spatial  relationships  of  the 
groups  In  three  dimensions.  We  feel  that  such  groupings  can  point  out 
the  characteristics  of  the  people  to  whom  a  questionnaire  is  given. 

These  characteristics  appear  to  be  useful  in  revising  the  questionnaire 
for  future  use. 

The  celling  height  weather  data  provided  us  with  the  opportunity 
to  Investigate  three  aspects  of  the  ISODATA-POINTS  process.  In  this 
sense  the  weather  problem  is  a  very  "rich"  problem  suggestive  of  many 
useful  modifications  of  ISODATA-POINTS.  In  particular  it  allowed  us: 

(1)  To  evaluate  its  ability  to  predict  ceiling  height; 

(2)  To  evaluate  its  capabilities  for  measurement  selection  ; 

(3)  To  le&rn  how  ISODATA-POINTS  exhibits  the  structure  of 
experimentally-obtained  data. 

The  performance  obtained  by  the  technique  was  slightly  better  than 
persistence  forecasting.  (Persistence  forecasting  is  the  technique  of 
forecasting  that  predicts  that  the  same  conditions  that  exist  at  the 
present  time  will  be  in  effect  at  some  later  time — (in  this  case  five 
hours).  This  prediction  was,  however,  made  without  utilizing  the 
categorization  information.  In  the  near  future  we  hope  to  improve  the 
prediction  score  by  utilizing  the  actual  ceiling  height  that  occurred 
five  hours  later  for  a  preliminary  sorting  of  the  patterns. 


The  analysis  of  this  data  was  supported  by  contracts  from  the  Behavioral 
Science  Division  of  the  Air  Force  Office  of  Scientific  Research. 

** 

By  measurement  selection  we  mean  the  determination  of  those  predictors 
or  measurements  that  are  most  useful  in  discriminating  between  classes. 
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Utilization  of  the  typical  patterns,  or  cluster  centers,  as  we 
have  called  them,  allows  us  to  evaluate  the  measurements  that  define 
each  of  the  patterns.  We  have  done  this  for  the  weather  data  and  have 
obtained  agreement  with  hr.th  th»  intnit(.-o  rzttnjr  cf  a  meteorologist 
and  measurement  evaluations  obtained  by  statistical  techniques.  We 
have  been  able  to  go  somewhat  further  than  this  in  one  respect.  From 
our  examination  of  the  Jaia  it  is  evident  tnat  when  one  considers  only 
the  weather  records  that  resulted  In  low  celling  heights,  the  important 
measurements  are  different  from  those  considered  important  when  one  uses 
high  ceiling  height  records  as  well.  This  indicated  to  us  that  predictors 
or  measurements  that  may  be  essential  in  one  region  of  pattern  space  need 
not  be  even  useful  at  all  in  other  regions.  Any  over-all  statement 
regarding  predictor  worth  that  averages  together  performance  in  different 
regions  of  pattern  space  seems  destined  to  obscure  such  important  details. 

By  plotting  the  cluster  centers  in  a  plane  using  the  distances  of 
the  clusters  from  each  other,  we  were  able  to  see  the  structure  of  this 
experimental  data.  This  wc  found  quite  suggestive  of  new  measurements 
that  should  be  made. 


VI  HOW  THE  OUTPUT  FROM  AN  ISODATA -PC I NTS  ANALYSIS  CAN  BE  USED 

The  information  supplied  by  an  I SODA TA- POINTS  clustering  consists 

of: 

1.  For  eech  cluster: 

a)  The  number  of  patterns  in  it; 

b)  The  average  distance  of  the  patterns  in  that  cluster 
from  the  average  point  of  the  cluster; 

c)  The  number  of  patterns  from  each  class  that  are  in  this 
cluster; 

d)  The  identity  of  the  patterns  that  are  in  that  cluster. 

2.  The  positions  of  a  set  of  average  points  that  the  process  has 
located  in  regions  of  high  pattern  density,  and  the  standard 
deviation  of  the  patterns  around  these  average  points  for  each 
of  the  pattern  components  • 

3.  The  distance  of  all  patterns  from  all  of  the  final  average 
points . 

4.  The  distance  of  each  average  point  from  every  other  average 
point,  i.e,,  the  distances  between  all  possible  pairs  of 
average  points. 

5.  The  average  distance  (taken  over  all  patterns)  from  a  pattern 
to  its  closest  cluster  point. 
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Using  this  information  it  is  possible  to  learn  a  great  deal  about 
the  structure  of  the  patterns  in  pattern  space.  The  gross  structure  of 
the  data  is  obtained  by  examining  the  spatial  relationships  between  the 
average  points.  Vote  *h?.t  the  r.urr.bw  average  points  _ls  small  enough 
to  allow  comparison  of  each  average  point  with  every  other  average  point. 

We  have  found  that  a  most  useful  way  of  comparing  these  average  point®  i® 
a  graphical  plot.  It  is  not  possible  to  draw  the  plot  in  the  original 
pattern  space  because  it  has  too  many  dimensions.  However,  by  using  the 
distances  between  pairs  of  average  points,  it  is  possible  to  plot  at 
least  three  average  points  on  a  flat  surface.  We  have  found  that  with 
real  data  we  have  frequently  been  able  to  plot  on  a  plane  more  than  three 
average  points  with  sufficient  accuracy  to  aid  our  intuition.  The  dis¬ 
tribution  of  patterns  around  these  average  points  can  be  plotted  using 
distances  from  these  average  points  and  a  more  detailed  understanding  of 
the  fine  structure  of  the  data  obtained. 

Using  the  Information  now  available  in  the  program  some  evaluation 
of  the  significance  of  a  given  clustering  is  possj_ble.  One  criterion  of 
clustering  that  can  be  used  is  average  distance  (AO)  of  a  pattern  from  its 
closest  average  point  (the  average  point  for_the  cluster  to  which  a  pattern 
belongs).  In  Fig.  22  we  show  the  value  of  AD  vs. the  number  of  clusters 
for  the  two-dimensional  example  as  the  iterations  progressed,  Note  that 
after  the  fourth  iteration  it  changes  little  as  the  number  of  masks  changes 
This  criterion  Is,  however,  probably  not  as  sensitive  as  might  be  desirod. 
By  using  the  distances  between  average  points  as  well  it  appears  possible 
to  determine  if  the  patterns  are  compactly  clustered. 

We  are  continuing  to  seek  new  ways  in  which  the  results  of  the  cluster¬ 
ing  can  be  analysed. 


VII  SUGGESTIONS  FOR  FURTHER  RESEARCH 

The  following  further  research  is  suggested  by  the  work  thus  far. 
In  addition  to  research  on  algorithms  for  ISODATA-LINIS  and  1S0DATA- 
PLA.NES ,  we  intend  to  investigate: 

(1)  Criteria  for  clustering,  in  order  to  improve  our  ability 

to  interpret  the  results  of  clustering  and  to  facilitate 
a  more  efficient  manipulation  of  the  process  parameters. 


(2)  Methodology  for  using  clusters  of  patterns  Here  we 
seek  methods  of  displaying  and  analyzing  the  results 
of  clustering. 

(3)  Classification  techniques  based  on  ISODATA-POINTS , 
IS0DATA-LIN88  «n<i  IROATA-PLANFS  that  use  distance 
from  points,  line  segments,  and  planar  segments  as 
the  criteria  for  determining  the  class  membership  of 
an  unknown  pattern. 

(4)  Actual  hardware  implementations  of  any  methods  that 
prove  promising  after  thorough  investigation  by  computer 
programs.  These  implementations  would  be  used  for  data 
analysis  and  classification  on  the  basis  of  distance 
from  points,  lines,  and  planes. 

(5)  Applications  using  computer  programs  Implementing 
ISODATA-POINTS ,  1S0DATA-LINES,  and  ISODATA- PLANES  on 
real-world  problems. 

In  the  following  five  sections  we  shall  discuss  these  areas  for 
further  research  In  aome  detail. 


1.  Criteria  for  Clustering 

So  far,  in  developing  the  ISODATA  techniques  we  have  contented 
ourselves  with  using  intuitively  satisfying  criteria  in  the  "decision- 
making"  in  the  computer  program.  At  this  time,  we  feel  that  we  should 
investigate  additional  analytical  juetif lcation  and  poaalbly  entirely 
new  criteria.  The  needed  criteria  are: 

(1)  Criteria  that  could  help  determine  the  "goodness  of 
fit"  of  a  given  clustering.  These  criteria  would  help 
define  "convergence"  for  ISODATA-llke  procedures  that 
learn  without  a  teacher. 

(2)  Criteria  for  lumping  and  splitting  of  the  clutters 

One  Important  aspect  of  this  part  of  the  work  le  the  deter¬ 
mination  of  the  effect  of  changing  the  scaling  function  used  for  various 
measurements,  e.g.,  changing  from  linear  to  logarithmic  scales.  Thia 
will  have  an  effect  on  the  clusters  found.  We  need  to  know  more  about 
the  extent  of  the  effect. 

There  exist  interesting  statistical  problems  in  this  work. 

For  example,  Dr.  Charles  Dawson  of  SRI  has  been  able  to  show  that  the 
sum  of  an  Infinite  set  of  n-dimenslonal  multivatiate  Gaussian  distri¬ 
butions  having  means  distributed  uniformly  along  a  straight  line  segment 
can  be  considered  as  n-1  dimensional  distributions  lying  in  hyperplanes 
having  the  straight  line  as  their  normal  vector,  except  very  close  to 
the  end  of  the  line  segment.  This  model  seems  an  interesting  one  for  the 
case  of  detecting  a  known  signal  with  time-varying  amplitude.  It  also  is 
quite  close  to  s  model  for  one  ISODATA-L1NES  cluster. 


2.  Methodology  for  Utilizing  Clusters  c f  Patterns 

In  our  work  thus  far  we  have  developed  several  methods  and 
prc^r-r.o  iv,  aid  in  accxn^  me  ii'ie  structure  ox  data  alter  clustering. 
Two  Important  ones  are: 

(1)  The  cluster  center  plot .  By  using  the  distance  between 
cluster  centers  we  are  able  to  plot  the  relative 
positions  of  the  cluster  centers  on  a  plane.  We  can 
always  plot  three  such  centers  and  still  satisfy  the 
inter-point  distance  constraints  exactly.  Frequently 
we  have  found  it  possible  to  plot  more  than  three  on 

a  plane  The  exact  number  that  it  is  possible  to  plot 
depends  on  the  spatial  relationships  that  exist  in  the 
data 

(2)  A  diatance-f rom-clus ter- center  computer  program.  Using 
this  program  we  axe  able  to  obtain  a  histogram  of  the 
distances  of  the  patterns  from  the  various  cluster 
centers.  This  gives  indications  of  the  distribution 

of  the  patterns  about  the  cluster  centers— i.e.  are 
they  loose  or  tight  clusters,  etc.  This  particular 
program  la  useful  in  setting  thresholds  and  weighting 
distances  between  pairs  of  clusters  for  the  classification 
of  patterns. 

Tt  is  essential  that  we  develop  other  methods  of  rapidly 
manipulating  these  clusters  of  data  in  order  to  learn  various  things 
about  the  fine  structure  of  the  patterns.  We  have  found  that  the  ideas 
come  most  easily  in  attempting  to  analyze  real  data.  We  are  particularly 
interested  in  drawing  together  these  techniques  to  develop  a  coherent 
methodology, 

3 •  Classlf lcation  Using  Distance  from  Lines  and  Planes 

ISODATA-POINTS  can  be  used  as  a  mode-seoklng  classification 
technique.  Our  work  on  the  development  of  ISODATA-POINTS  has  also 
helped  us  understand  so-called  piecewise-linear  error-correction  classi¬ 
fication  techniques.  it  therefore  stums  plausible  that  :!>o  development 
of  I SODATA- LINE S  and  ISODATA-PLANES  should  help  the  development  of 
classification  techniques  that  are  based  on  the  distance  from  a  set  of 
line  segments  or  from  a  set  of  planar  segments,  where  different  sets  of 
line  segments  for  example,  would  be  associated  with  the  different  classes 
(see  Fig.  24) . 

For  certain  classes  of  patterns,  such  as  speech,  word  recogni¬ 
tion,  or  speaker  recognition,  and  other  non-stat ionary  time-series  analy¬ 
sis  problems,  this  type  of  technique  may  prove  quite  powerful  Another 
such  application  might  be  optical  pattern  recognition — specifically  with 

* 

This  seems  all  the  more  true  when  the  possibility  of  optical  implementa¬ 
tion  exists  and  makes  pattern  dimensions  of  1000  reasonable. 
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FIG.  24  CLASSIFICATION  OF  PATTERNS  BASED  ON  DISTANCE 
FROM  PIECEWISE-LINEAR  CURVES 
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FIG.  25(a)  IMPLEMENTATION  FOR  FINDING  EUCLIDEAN  DISTANCE 
OF  A  PATTERN  FROM  A  POINT 
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FIG.  25(b)  IMPLEMENTATION  FOR  FINDING  EUCLIDEAN  DISTANCE 
OF  A  PATTERN  FROM  A  LINE 
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FIG.  25(c)  IMPLEMENTATION  FOR  FINDING  EUCLIDEAN  DISTANCE 
OF  A  PATTERN  FROM  A  PLANE 


ITS  INTENSITY  INDICATES  VALUE  OF  VARIABLE  I. 


FIG.  26  AN  OPTICAL  PANEL  FOR  INPUTTING  A  HIGH-DIMENSIONAL  PATTERN 
INTO  AN  ISODATA  SYSTEM 
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We  are  attempting  to  improve  and  refine  our  techniques  for 
measurement  selection  and  to  look  at  the  possibilities  of  generating 
meaningful  measurements 
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APPENDIX  A 


AND  I  SODA  TA— PLANES 

In  thi««  appsndix  wc  cor»5  i  uOr  ihv  generalization  of  ISODATA- PCI  NTS 
to  the  fitting  of  connected  line  segments  to  "tubular"  high  dimensional 
data  and  to  the  fitting  of  triangulares egments  of  hyper-planes  to 
"surface-like"  high-dimensional  data. 

The  fundamental  concept  of  ISODATA  is  the  iterative  adjustment  of 
the  position  of  "clusters"  in  order  that  these  clusters  come  to  reflect, 
in  their  relative  positions,  the  structure  of  the  data.  In  ISODATA- POINTS 
the  goal  is  to  adjust  these  clusters  so  that  they  lie  around  well-chosen 
average  points. 

In  the  generalizations  of  I SOnATA- POINTS  in  this  section  we  will 
relate  first  pairs  (ISODATA-LINES)  and  then  triples  (ISODATA- PLANES)  of 
points  to  each  other.  In  the  first  generalization  we  relate  pairs  of 
points  together  in  order  to  oreste  line  segments;-  By  allowing  each  point 
to  be  in  more  than  one  pair  we  are  able  to  create  a  piecewise-linear 
curve,;  We  propose  fitting  curves  composed  of  segmsnts  of  these  lines  to 
the  data  (rather  than  jUst.  single  points) ,  thus  obtaining  an  piece-wise 
linear  algebraic  expression  describing. a  set  of  data  points  ins  high- 
dimencional  space.  Such  iterative  .fitting  of  a  set  of  line  segments  to 
data  we  call  ISODATA- LINKS.  Such  a  curve  is  shown  fitted  to  s  set  of 
hypothetical  data  in  Fig.  A-l 

In  the  second  generalization  of  ISODATA- POINTS  we  associate  triples 
of  points  together.  These  triples  of  points  can  be  used  to  define  a 
triangular  segment  of  a  plane,  inn-dimensional' space.  By  allowing  points 
to  be  in  more  than  one  triple  of  points  we  Jink  these  triangular  segments 
of  planea  together  to  form  a  piecewise-plansr  Surface  in  n-space.  This 
surface  would  then  be  iteratively  adjusted  to  cause  It  to  fit  a  set  of 
data.  This  technique  we  call  I SODA TA- PLANES  Such  a  surface  in  a  three- 
dimensional  space  is  shown  in  Fig.  A-2. 

We  have-not  developed  an  adequate  algorithm  for  either  of  these 
general izationa .  We  have  investigated  ISODATA-LINES  to  a  greater  extent, 
the  results  of  which  we  now  describe. 


An  example  of  "tubular"  data  would  be  a  set  of  time  samples  of  the 
patterns  at  the  output  of  a  set  of  band-pass  filters  into  which  a  word 
has  been  spoken.  An  example  of  surface-like  data  is  the  values  of 
n-predictors  that  are  used  to  predict  a  single  predictand  such  as  the 
ceiling  height  at  Washington,  D.C. 
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A.  ISODATA- LIKES 


At  the  present  time  it  appears  to  us  that  the  algorithm  fc; 
ISCD.VTA- LIKES  snouia  consist  of  the  following  steps: 

(1)  Use  ISODATA-POINT5  to  define  cluster  centers  within  the 
data . 

(2)  Take  the  nearest  two  cluster  points  and  rebate  them 
(le.,  they  are  to  define  a  line  segment). 

(3)  Starting  with  one  cluster  point  of  this  pair,  find  that 
new  cluster  point  nearest  to  it.  Relate  this  pair. 

(A  maximum  allowable  distance  for  pairs  might  be  used 
here.) 

** 

(4)  Continue  this  procedure  until  all  acceptable  cluster 
points  are  paired.  At  this  point  Iterative  adjustment 
of  the  line  segments  would  begin. 

(8)  Effective  iterative  adjustment  of  the  line  segments 
requires  the  answering  of  the  following  two  questions: 

(a)  What  subset  of  patterns  should  be  assooiated 
(probably  not  dis jointly)  with  each  line  segment? 

(b)  In  what  direction  should  eaoh  pattern  move  the 
cluster  points  that  define  the  line  segment 
assooiated  with  that  pattern  and  what  amount  should 
it  move  it? 


Bach  pair  considered  for  a  relationship  should  be  examined  to  ensure 
that  there  are  points  lying  near  the  straight  line  segment  connecting 
them.  A  simple  modification  of  the  I 80DATA- POINTS  program  to  store 
the  second  (and  third?)  cluster  points  nearest  to  the  patterns  would 
allow  the  presence  or  absence  of  patterns  between  two  cluster  points 
to  be  found. 


Some  cluster  points  may  bs  isolated  from  others  due  to  the  nature  of 
the  data  and  in  thla  sense  "unacceptable. "  If  this  isolated  point  were 
aplit  into  two  points  in  the  manner  of  XSOQATA-POXNTS  a  beet-fit  line 
to  thia  isolated  cluster  could  be  obtained. 


A-4 


These  are  two  reasonably  well-defined  questions.  Though 
we  have  no  tested  definitive  answers,  we  have  the  following 
conjectural  answers: 

To  5(a);  A  pattern  should  be  associated  with  the  two 
line  segments  to  which  it  is  closest. 

To  5(b);  A  pattern  should  move  the  cluster  points  in  a 
direction  toward  the  pattern  in  a  direction 
perpendicular  to  the  line  connecting  the  extreme 
end  points  of  two  line  segments  sharing  a  common 
center  cluster  point.  In  Fig.  A-3  this  line  is 
shown  as  a  dashed  line  connecting  Cluster  Point  1 
and  Cluster  Point  3. 

The  details  of  the  proposed  adjustment  procedure  (for  two 
dimensions)  are  shown  in  Fig.  A-4 .  A  perpendicular  BD  is,  in  effect, 
erected  to  line  AC.  Patterns  associated  with  these  two  line  segments 
whose  projection  (on  the  line  AC) are  in  the  interval  AD  are  used  to 
modify  Points  A  and  B.  Patterns  in  the  interval  DC  are  used  to  modify 
Points  B  and  C.  A  pattern  projecting  directly  on  A  would  modify  only 
A,  and  the  same  for  C.  A  pattern  projecting  on  D  would  modify  only  B. 
The  proportion  of  modification  made  to  each  of  the  two  points  for  cases 
in  between  would  be  linear,  as  indicated  in  Fig.  A-4(b) .  In  Fig.  A-4 (c) 
we  show  the  amount  of  modification  made  for  a  sample  pattern  projected 
onto  AD. 


Note  that  the  bias  in  this  adjustment  procedure  tends  to 
straighten  the  kinks  in  the  piecewise-linear  curve  shown  in  Fig.  A-3. 

As  we  develop  a  better  understanding  of  ISODATA-LINES ,  it  seems 
nearly  certain  that  something  analogous  to  splitting  and  lumping  will  be 
useful.  In  Fig.  A-5  we  indicate  two  situations  in  which  different  kinds 
of  splitting  might  be  called  for.  As  for  lumping,  it  could  conceivably 
occur  if  two  cluster  points  draw  close  together  (i.e.,  as  the  line  segment 
between  them  shortens)  or  as  a  cluster  point  at  the  end  of  a  curve  draws 
near  to  a  line  segment . (Some  provision  for  branching  data  would  be 
necessary.)  Straightness  of  consecutive  segments  could  also  be  used  as 
a  criterion  for  dropping  the  interior  cluster  point  (this  is  the  reverse 
of  the  splitting  in  Fig.  A-5(b)). 

We  feel  that  the  use  of  the  ISODATA- POINTS  program  to  find 
reasonable  starting  lines  will  greatly  reduce  running  time  by  reducing 
the  number  of  iterations  required  by  ISODATA-LINES  to  find  a  good  fit 
to  the  data. 


The  calculation  of  distance  from  a  line  segment  does  not  require  more 
than  the  algebraic  manipulation  of  the  distances  from  the  two  defining 
end  points.  No  explicit  formula  for  the  line  segment  itself  need  be 
found. 
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FIG.  A-4(o)  THE  SUBSET  OF  PATTERNS  ASSOCIATED  WITH  TWO  LINE  SEGMENTS 

(b)  THE  CLUSTER  POINTS  AFFECTED  BY  PARTICULAR  PATTERNS 

(c)  THE  PROPORTION  OF  CLUSTER  POINT  MODIFICATION 
CAUSED  BY  A  SINGLE  PATTERN 


B.  I SODA TA- PLANES 


We  have  not  yet  workec  out  a  tentative  algorithm  for  the  Itera¬ 
tive  adjustment  of  these  planes,  except  insofar  aa  the  algorithm  la 
similar  to  isooATA-LINES .  We  feel  that  by  the  time  we  have  developed 
ISODATA- LINES  we  will  have  a  good  start  on  developing  ISODATA- PLANES. 

It  is  Interesting  to  note  that  for  ISOn*TA-PLANES  we  only  need  distances 
from  the  three  defining  points  in  order  to  find  the  distances  from  a 
plane.  We  do  not  need  an  explicit  formula  for  the  plane. 
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